Update Dask/Distributed version by pentschev · Pull Request #3978 · NVIDIA/cuml

pentschev · 2021-06-10T20:02:23Z

No description provided.

dantegd · 2021-06-10T20:28:50Z

dantegd · 2021-06-10T22:26:29Z

@ajschmidt8 @pentschev CI seems to be installing an older nightly of dask-cudf:

dask-cudf                 21.08.00a210609 py38_g83bc0320b7_125    rapidsai-nightly

which probably is the reason we're getting these fails?

jakirkham · 2021-06-10T22:33:48Z

Guessing this is relevant ( rapidsai/cugraph#1637 (comment) )

dantegd · 2021-06-11T02:24:26Z

rerun tests

jakirkham · 2021-06-11T12:50:58Z

rerun tests

dantegd

Why the change

dantegd · 2021-06-11T15:59:02Z

+- dask>=2021.6.0
+- distributed>=2021.6.0


Why change from installing from main? Since CI is installing from main, we should keep this so dev environments behave like CI (and mirror cudf https://github.com/rapidsai/cudf/blob/0a4e8a1c860518c9e30f5289dea07bd0879fb642/conda/environments/cudf_dev_cuda11.2.yml#L64)

That's still done above. Peter is just setting a minimum version

No, I think what Dante was asking for is that the conda dev environment files install Dask/Distributed main to match the same as CI is running on. I did that in 80c340d now.

pentschev · 2021-06-11T20:52:31Z

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:

ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in NVIDIA/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Also FYI, I'll be on PTO for the next couple weeks, so maybe someone will need to take this over or ensure those issues are addressed on a different PR before merging this.

galipremsagar · 2021-06-11T21:15:54Z

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:
ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.
I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in rapidsai/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Just looked at the errors, this bug is in dask-cudf which is introduced by : NVIDIA/cudf#8309, I can throw in a fix for this real quick.

pentschev · 2021-06-11T21:23:09Z

Just looked at the errors, this bug is in dask-cudf which is introduced by : rapidsai/cudf#8309, I can throw in a fix for this real quick.

That would be great, thanks @galipremsagar !

galipremsagar · 2021-06-12T01:11:51Z

I've checked out cugraph(cugraph CI is also having a similar issue: rapidsai/cugraph#1637 (comment)) locally and built from source with latest dask and don't find the errors that I see in CI. This looks like somehow the older version of dask is not being uninstalled correctly in those images. I see that tests in the jobs that are currently running are passing, so will keep an eye out on this one.

dantegd · 2021-06-12T01:29:17Z

rerun tests

dantegd · 2021-06-12T01:29:24Z

@gpucibot merge

galipremsagar · 2021-06-12T03:13:15Z

rerun tests

jakirkham · 2021-06-12T03:53:30Z

rerun tests

jakirkham · 2021-06-12T07:16:41Z

rerun tests

jakirkham · 2021-06-12T08:38:02Z

rerun tests

jakirkham · 2021-06-12T15:51:38Z

rerun tests

codecov-commenter · 2021-06-12T18:13:28Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@8fe1b05). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-21.08    #3978   +/-   ##
===============================================
  Coverage                ?   85.32%           
===============================================
  Files                   ?      230           
  Lines                   ?    18093           
  Branches                ?        0           
===============================================
  Hits                    ?    15437           
  Misses                  ?     2656           
  Partials                ?        0

Flag	Coverage Δ
dask	`47.90% <0.00%> (?)`
non-dask	`77.66% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8fe1b05...80c340d. Read the comment docs.

galipremsagar · 2021-06-14T14:48:08Z

rerun tests

Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - https://github.com/jakirkham - Dante Gama Dessavre (https://github.com/dantegd) URL: NVIDIA#3978

pentschev added 2 commits June 10, 2021 12:58

Install Dask/Distributed main in CI

6fb00fb

Install Dask/Distributed 2021.6.0 in conda environments

e541896

pentschev requested a review from a team as a code owner June 10, 2021 20:02

github-actions Bot added conda conda issue gpuCI gpuCI issue labels Jun 10, 2021

ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jun 10, 2021

ajschmidt8 approved these changes Jun 10, 2021

View reviewed changes

jakirkham approved these changes Jun 10, 2021

View reviewed changes

dantegd approved these changes Jun 10, 2021

View reviewed changes

dantegd requested changes Jun 11, 2021

View reviewed changes

Use Dask/Distributed main in conda dev environments

80c340d

pentschev mentioned this pull request Jun 11, 2021

Update to changed rmm::device_scalar API rapidsai/cugraph#1637

Merged

dantegd approved these changes Jun 12, 2021

View reviewed changes

rapids-bot Bot merged commit 4f13873 into NVIDIA:branch-21.08 Jun 14, 2021

pentschev deleted the update-dask-versions branch June 28, 2021 10:59

Uh oh!

Conversation

pentschev commented Jun 10, 2021

Uh oh!

dantegd commented Jun 10, 2021

Uh oh!

dantegd commented Jun 10, 2021

Uh oh!

jakirkham commented Jun 10, 2021

Uh oh!

dantegd commented Jun 11, 2021

Uh oh!

jakirkham commented Jun 11, 2021

Uh oh!

dantegd left a comment

Choose a reason for hiding this comment

Uh oh!

dantegd Jun 11, 2021

Choose a reason for hiding this comment

Uh oh!

jakirkham Jun 11, 2021

Choose a reason for hiding this comment

Uh oh!

pentschev Jun 11, 2021

Choose a reason for hiding this comment

Uh oh!

pentschev commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

galipremsagar commented Jun 11, 2021

Uh oh!

pentschev commented Jun 11, 2021

Uh oh!

galipremsagar commented Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dantegd commented Jun 12, 2021

Uh oh!

dantegd commented Jun 12, 2021

Uh oh!

galipremsagar commented Jun 12, 2021

Uh oh!

jakirkham commented Jun 12, 2021

Uh oh!

jakirkham commented Jun 12, 2021

Uh oh!

jakirkham commented Jun 12, 2021

Uh oh!

jakirkham commented Jun 12, 2021

Uh oh!

codecov-commenter commented Jun 12, 2021

Codecov Report

Uh oh!

galipremsagar commented Jun 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pentschev commented Jun 11, 2021 •

edited

Loading

galipremsagar commented Jun 12, 2021 •

edited

Loading